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(57) Abstract: 'llic invention relates to a video coding 
method based on a subband coding scheme applied to 
a secjuence of two-dimensional frames. Said method 
comprises a subband decomposition step of a current frame 
(F2), a motion prediction step, carried out with respect 
to a previous or reference frame (I'D, and a coding step. 
According to the invention, the motion prediction step is 
based on a redundant decomposition of the reference frame, 
according to Ihe Ibllowing procedure : in the approximation 
subband, ihc motion is estimated by means of a full search 
operation applied to each pixel of the current frame, and 
for the other subbands, the motion is estimated by means 
of a pel-recursive motion estimation algorithm using for the 
prediction the reference redundant subband reconstructed 
from the subsampled subbands. Said pel-recursive motion 
estimation algorithm itself comprises an initialixation 
sub-step, a determination and updating sub-step, and a 
decision sub-step. 
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Video coding method 



FIELD OF THE INVENTION 

The present invention generally relates to the field of video compression and, 
more specifically, to a video coding method based on a subband coding scheme applied to a 
sequence of two-dimensional frames, said method comprising a subband decomposition step 
5 of a current frame (F2), a motion prediction step, carried out with respect to a previous or 
reference frame (Fl), and a coding step. 

BACKGROUND OF THE INVENTION 

In the multimedia domain, new interactive applications such as Internet video 

10 streaming, video database browsing or multi-quality video services, are becoming 

widespread. This recent expansion of video services over heterogeneous networks (Internet, 
mobile nets, In Home Digital Networks) has raised new issues in terms of varying transport 
conditions (bandwidth, error rate. . .) as well as varying consumer demands and terminal 
decoding capacibilities (CPU, display size, application. . .), and has led to investigate new 

15 algorithms for video compression, in particular methods based on subband decompositions. 

In the conventional video coding algorithms, and more particularly within the 
frame of the MPEG-4 standard, the motion estimation between successive frames of a 
processed video sequence is carried out by means of the so-called block-matching algorithm 
(BMA) : in BMA, a motion vector is assigned to a block of picture elements (pixels) in a 

20 current frame (decomposed into blocks of fixed size) by searching a similar block within a 
defined area - or search window - of a reference frame, the best vector (that represents the 
shifting of the block) being found by evaluating candidates according to an error measure 
(the best corresponding block is the one that gives rise to a minimal error). The BMA 
unfortunately generates high frequency artefacts at the blocks edges (blocking effects) in the 

25 motion compensated frames. When the BMA is used in coding schemes based on a wavelet 
decomposition, these artefacts reduce the coding efficiency by inducing high coefficients in 
the wavelet decomposition of the compensated firames. 



SUMMARY OF THE INVENTION 
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The object of the invention is to propose a video coding method incorporating 
another type of motion estimation and compensation and allowing to improve the coding 
efficiency. 

To this end, the invention relates to a method such as defined in the 
5 introductory paragraph of the description and which is moreover characterized in that said 
motion prediction step is based on a redundant decomposition of the reference frame, 
according to the following procedure : 

(1) in the approximation subband of said decomposition, the motion is 
estimated by means of a full search operation involving said reference frame and applied to 

10 each picture element (pixel) of the current frame ; 

(2) for the other subbands, the motion is estimated by means of a pixel (or pel) 
recursive motion estimation algorithm using for the prediction the reference redundant 
subband reconstructed from the subsampled subbands yielded by said subband 
decomposition. 

IS The proposed technical solution is based on the implementation of a pel- 

recursive motion estimation algorithm using wavelets. A motion estimation in the wavelet 
domain has to cope with the problem of translation invariance. The redundant decomposition 
of the reference frame is then used to predict the motion, allowing thus to take into account 
motion on the finest resolution grill. In the approximation subband, the motion is estimated 

20 by a full search algorithm applied to every pixel in the current frame. For the other subbands, 
a pel-recursive algorithm using for prediction the reference redundant subband is 
implemented. The initialization of this algorithm is made using a weighted value of the 
motion vectors of the spatial neighbours in the same subband and of the motion vector 
corresponding to the same position at the previous level of the decomposition. The algorithm 

25 allows to do a re-initialization at the positions failing to converge towards a good estimate. 
The scanning order in the subbands is also optimized in order to minimize the drift that can 
occur in a line-by-line scan. 

More specifically, said pel-recursive motion estimation algorithm may 
comprise the following sub-steps: 

30 (a) an initialization sub-step, the iterative algorithm being initialized with the 

mean value do (m) of the motion vector in a causal neighbourhood S : 

the following notations being used : 
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m = pixel for which the motion has to be estimated 
d(m) = its motion vector ; 

(b) an updating sub-step, for updating di according to the 

relation : 

5 di+i = di + Ui, where Ui, determined at each iteration i, is an update vector given by : 

where V A is the gradient of the image at the point m-di, E(m) = B(m) - A(m-di) is the 
prediction error at the current iteration, and A is a regulaiization parameter ; 

(c) a decision sub-step, for carrying a break test and thus ending the procedure 
10 if dj is close enough to d ; 

a typical neighbourhood of four pixels being preferably used. 

In a particular inq>lementation of the invention, the break test may be based 
on the estimation of die following ratio : 



R = 



.1' 



15 said ratio having to be greater than a given threshold £ . 

Said break test may also include an additional condition put on a maximum 
number of iterations. 

It may also be indicated that, according to an advantageous embodiment of the 
invention, the determination of said update vector Ui is based on a computation step including 

T 

20 the following minimizing operation : knowing that B(m) = A(m-di) • Si .VA where 

Ai^.VA is the inner product of VA and vector Si = d - dj, to minimize the square error J : 
J = (B(m) - A(m-di) + Ai'^.VA)^ + A||&l|p 

with respect to Si , the term ^jSif' being a regularization term provided in view of a trade- 
off between the smoothness of the resulting motion vector field, for large values of A , and 
25 the accuracy of the motion vectors, for small values of A , the minimum being obtained for 
dJ/dSi = 0 and leading to the update vector : 

||VAf+X 
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In another embodiment of the invention, the motion estimation algorithm may 
also comprise the following sub-steps : 

(a) an initialization sub-step, the iterative algorithm being initialized with the 
mean value dj^ (m) of the motion vector in a causal neighbourhood S: 



20 



Card(S)j5 

the following notations being used : 
k G s (1 to 4) 

m = pixel for which the motion has to be estimated 
dj^o(ni) = its motion vector ; 
10 (b) an updating sub-step, for updating dj, m according to the 

relation : 

di, i+i = djj + Uj,i, where Uj,i determined at each iteration i is an update vector given by : 

=( E(VAj,sVA];s)+;1D-' (- EEj.s(m)VAj.,) 
s=04 s=0,l 

where : 



15 VA,.,, = 



dm 

dAy^(in-djj) 

ail 



is the gradient of the image at point m - djj 
and Ei.s (m) = Bj^ (ra) - Aj^ (m - dj,i) 

(c) a decision sub-step, for carrying a break test and thus ending the procedure 
if dj j is close enough to dj. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will now be described, by way of example, with 
reference to the accompanying drawings in which: 

Fig. 1 illustrates a subband decomposition of a video sequence, with motion 
25 compensation ; 

Figs. 2 and 3 respectively show a dyadic subband decomposition and a bank 
filter allowing to obtain said decomposition ; 

Figs. 4 and 5 illustrate how the subsampling operation of such a dyadic 
subband decomposition does not preserve the translation invariance ; 
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Figs. 6 and 7 illustrate identities between representations, showing that 
redundant decompositions exist for each level of analysis; 

Figs. 8 and 9 depict the so-called lifting scheme, respectively in the case of a 
signal decomposition and in the case of a signal reconstruction; 
5 Fig. 10 to 12 show three levels of the classical wavelet decomposition 

obtained by applying the identities of Figs.6 and 7 to each bank filter of Fig.3; 

Fig. 13 illustrates the choice of the samples (0 for even, 1 for odd) for a 
decomposition on three levels ; 

Fig. 14 shows the 64 possible subsampling choices corresponding to the 64 
10 possible decomposition bases; 

Fig. 15 shows how interleaving subsampled subbands yields the redundant 

subband ; 

' Fig. 16 illustrates the redundant decomposition of a single frame, everyone of 
the ten redundant subbands having the same size as the original firame ; 
IS Fig. 17 shows the classical, linear scanning order, and Fig. 18 a causal 

scanning order allowing to obtain an improved causality in the initialization of the described 
pel-recursive algorithm ; 

Fig. 19 shows a fiill search window in the redundant plane for the pixel (m,n). 

20 DETAILED DESCaWPTION OF THE INVENTION 

In order to obtain flexible video coding systems, able to cope with different 
requirements and capabilities, the scalability is the expected functionality. Progressive 
encoding techniques based on subband decompositions may be an answer, since they allow a 
fully progressive transmission. Particularly, wavelets have a high efficiency in progressively 

25 encoding images in view of a scalable representation. They offer a natural multiscale 

representation for still images, that can be extended to video data by means of a 3D (or 2D+t) 
wavelet analysis including the temporal dimension within the decomposition (3D = three- 
dimensional ; 2D = two-dimensional ; t = time). The introduction of a motion compensation 
step in the 3D subband decomposition scheme leads to a spatio-temporal multiresolution 

30 (hierarchical) representation of the video signal, as illustrated in Fig.l showing a temporal 
subband decomposition of the video information, with motion compensation. 

The illustrated 3D wavelet decomposition with motion compensation is 
applied to a group of frames (GOF), referenced Fl to F8. In this 3D subband decomposition 
scheme, each GOF of the input video is first motion-compensated (MC) in order to process 
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sequences with large motion, and then temporally filtered (TF) using Haar wavelets. The 
dotted arrows correspond to a high-pass temporal filtering, while the other ones correspond to 
a low-pass temporal filtering, and three stages of decomposition are shown (L and H = first 
stage ; LL and LH = second stage ; LLL and LLH = third stage). 

However, when performing a pel-recursive motion estimation on the 
coefficients of each subband, ttie lack of translation invariance must be taken into account. 
The major problem with a dyadic subband decomposition (see Fig.2, showing such a 
decomposition, performed for three layers and leading to ten subbands LHl, HLl and HHl in 
the lowest layer. LH2, HL2 and HID in the middle layer, and M3, LH3, HL3 and HH3 in the 
highest layer ; and Fig.3, depicting a bank filter that allows to obtain this decomposition from 
an input signal f[n], giving output signals Lltn], L2[n], L3[n], Hl[nl, H2[n] and H3[n]) is 
indeed the fact that the subsampling operation (31, 32, 33, 34, 35, 36) applied after each 
filtering operation (21, 22, 23, 24, 25, 26) does not preserve the translation invariance (while 
the redundant wavelet transfomi of a signal is a time invariant representation, on the contrary 
the non redundant wavelet decomposition lacks this property). An example is given in Hgs 4 
and 5, whoe the samples in the high and low filtering branches, at the outputs of the low- 
pass and high-pass filters (L(z) andH(z)) and the subsamplers (™ 2), are shown respectively 
in the case of a first position of the edge of a moving object (original signal ORS) and in the 
case of a shifted input (shifted input SHE) corresponding to another position of said edge. 
The obtained signal is shown just after each filter and each subsampler. Each subsampling 
operation has to keep ather tiie even or the odd samples : a moving edge has resulted in a 
high frequency coefficient HFC in the first case (Fig.4, high filtering branch), while said edge 
is no longer detectable in the second one (Rg.5). Consequently, one cannot deduce motion 
from subsampled subband coefficients. The motion prediction must be performed from 

unsampled subbands. 

For a ID signal, a subsampling operation yields two possible 
choices : either odd or even samples can be taken. For a 2D signal (such as an image), this 
operation yields four possible decompositions. Thwefore, if a signal frame is decomposed 
upon three levels, 64 possible decompositions, equivalent in terms of quantity of information, 
can be obtained. However, as previously mentioned, a motion estimation cannot be 
performed on them, because they are not redundant. It is then observed from Fig.6 
(illustrating the identity between the two representations (L(z^) +™ 2) and C™ 2 + L(2) shown 
in tiiat figure) that redundant decompositions exist for each level of analysis. The 
introduction of Uie equivalence of Fig.6 in the low filtering branch (21, 31, 22, 32, 23, 33) of 
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Fig.3 (repeated in the upper part of Fig.7) leads to the equivalent scheme of the lower part of 
said Fig.7 : now, before the final subsampling operation, a redundant subband whose size is 
the same as that of the original image is obtained, and a valid motion estimation can be 
performed on it (the redundant subbands RSBl, RSB2, RSB3 at the three levels 1, 2, 3 have 
5 been shown in Fig.7). 

The wavelet decomposition itself is performed using the so-called lifting 
scheme, depicted for instance in Fig.8. in the case of the signal decomposition leading to 
outputs c[n] and d[n], and Fig.9, in the case of the signal reconstruction from said signals c[n] 
and d[n] (this implementation has been chosen because of the flexibility for the choice of the 
10 prediction filters P and update filters U, but it is obviously not the only one possible). For 
each filter pass, either odd or even samples (xi[n]. xo[n] respectively) can be taken first, as it 
can be seen in Figs.8 and 9 : 

- even samples first ? : 

Xo[n] = f[2n] ] 

x\>0 

Xi[n] = fI2n + l]J 
15 - odd samples first ? : 

Xo[n]=:f[2n+l] 1 
xi[nl = f[2(n + l)]j"" 

Figs. 10 to 12 then show the "classical" wavelet decomposition obtained by 
applying the identities of Figs.6 and 7 to each filter bank of Fig.3 : one can see that stopping 
just before the decimation at each level (Fig. 10, 1 1 or 12) yields the redundant subband. 

20 The problem is that lifting wavelets inherently use decimation before filtering. 

In Fig.8, where the input signals for the lifting bank are Xo[n] = fl2n].X|[n] = fT2n+l] or 
xo[n] = f[2n+l].xi[n] = f[2(n+l)], these inputs are decimated with respect to the original 
signal f[n]. Thus the decimation step cannot be skipped during the decomposition, and the 
redundant subbands can be obtained in another way, now described. 

25 It has been seen, in relation with Figs 8 and 9, that for each level of 

decomposition and for each dimension, it is possible to choose even or odd samples first. If 
for instance the even choice is represented by 0 and the odd one by 1. a full decomposition on 
three levels is then characterized by six bits, as shown in the example of Fig. 13 which can be 
seen as the binary decomposition of tiie number of the decomposition basis : 0,0,1,0,0,1 -> 

30 basis number 9. As a result, there are 64 possible subsampling choices corresponding to the 
64 possible decomposition bases (as illustrated in Fig. 14). 
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The critically subsampled subbands are then used to construct the redundant 
subband. The number of the decomposition basis yields the position where the subband will 
be placed on a rectangular grid. For example, if the bits describing x are 0 1 0 (3^^* level, 2"** 
level, 1^^ level), they yield the number 2, which is the reconstruction offset showing where the 
5 first sample of the subband has to be placed in the redundant subband. following the x 

direction. Hie same operation is applied in the y direction, and the samples are then placed 
at intervals of length : interval = 2^"^"^^°° Knowing the offset and the interval for each 
subsampled subband, it is possible to interleave them in order to reconstruct the redundant 
subband, as shown in Fig. IS. This redundant subband has the same size as the original image. 

10 At the first level of decomposition, each of the redundant subbands HL, LH and HH is 
obtained by interleaving four subsampled subbands. At level 2, they are obtained by 
interleaving 16 subsampled subbands, and at level 3, one has to interleave 64 subsampled 
subbands to obtain the redundant one (see Fig.l4). Eventually, the redundant decomposition 
yields ten redundant subbands, as illustrated in Fig. 16. 

IS The pel-recursive motion estimation will be now described. In what follows, 

bold characters for vectors (d), Vfor the gradient, and (.)^ for transposition will be used. The 
following algorithm is meant to compute the Opdc Flow between two matrices A and B , 
which can be two successive frames of a sequence, or (as considered in the present case) two 
successive subbands. The aim is to estimate for each pixel m = (m^n), the motion vector d(m) 

20 : 

5(m)=A(m-d) (1) 
To perform a recursive description of d, it has to be assumed that d is smoothly distributed 
over the image plane. Basically the method described here is : for each reference pixel in B , 
the gradient in A is computed at the estimated position (m-d) in order to find a pixel in A 

25 that is closer to the reference pixel. This yields a new position m-d. If it is the same position, 
the algorithm stops, else it iterates once more. Thus the algorithm converges and stops when 
it has reached the right position. 

The iterative procedure is composed of the following steps, including the 
initialization, the computation of an update vector, the update of the motion vector, and a 

30 break test : 

(1) The iterative procedure is initialized by the mean value of the motion 
vector in a causal neighbourhood S, 
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considering that a typical neighbourhood of four pixels can be used (die motion vectore are 
supposed to be zero out of the image boundaries) : 



* ji h }3 * 



* jo m * 



He 

* * * 



vector Ui 



where 



(2) The second step is the determination, at each iteration i, of an update 
E(m) 



Ui = 



/'a4(m-d|)^ 

dm 
aA(m-d|) 

dn 



-VA 



(3) 



(4) 



is the gradient of the image at point m-di, 

E(m)=: B(m)' A(m.di) (5) 
is the prediction moT at the current it^ation, and A is a legularization parameter. 

(3) In the third step, di is updated : 

di+i=di + ui (6) 

(4) Hie break test ends the procedure if di is close enough to d. This test can be 
for instance based on the estimation of the ratio : 



(7) 



which has to be superior to e (e can be turned to balance the vector determination precision 
and the computational load). A supplementary condition may be put on the maximum 
number of iterations. 

The second step (update vector computation, described by the relation (3)) can 
be detailed. The first order approximation of the relation (1) reads : 

B (m) = A (m-di) - Sd^.V A (8) 
where V A is the gradient from the relation (4) and ^ d = d - di 

One has to minimize the square error (the term >l||i5d|^ will be explained later) : 
min«( ()g(m) - A(m ■ dj ) + .VA)^ + A\\dAf ) (9) 
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dj 

The minimum is obtained when : — - = 0 , i.e. when : 

(B(m) - A(m -di ) + SA^.VAyVA + AAI = 0 (10) 

As Sd^.VA is the inner product of vectors 51 and , one has : iSi^.VA = VA*^. Si and 
the relation (10) becomes : 

(VA.V'^ + MlSd = -E(m).VA (11) 
The utility of /I can now be seen : the matrix VAVA^ is not invertible (rank 1). but 
VAVA^ 4- can be inverted using the following lemma : 

(M + MM^) ^=M* = r- 

which yields here for u = VA and M = /I/ : 

VA 

and finally leads to : 



10 ((VAVa'^+;1/)-^= (12) 



ja=~5^V^ (13) 



This is the update vector Uj used in the relations (3) and (6). This regularization provides a 
trade-off between the smoothness of the resulting motion vector field (for large A ) and the 

15 accuracy of the motion vectors (for small A ). 

It has been seen, in the relation (1 1), that VA has to be computed. The 
gradient computation can be approximated by a discrete filtering of the image. To perform a 
discrete approximation of the gradient by linear filtering, the filters must satisfy some 
conditions that will be now reminded. Assuming that h is a FIR filter, and considering the 

20 rectangular sampling of a continuous field F(x,y) , one has : F[n, m] = F(nL, , nLj ). The 
gradient of the image following the x direction is : 
dF 

Li •t:(»^1»w^) = Vw *F[n,mJ 
dx 

= Shk,lF[n-.k,m-l] (14), 
(k,l)6S ' 

S being considered as a close neighbourhood of 9 pixels (the center pixel m and the eight 
pixels around). On the other hand, at first order, one has : 
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F[n - k. m - 1] » F[n, m] - kLi.|^-(iiLi,nL2) - IL2.— («Li, nLj) (15) 

ox dy 

and combining the relations (14) and (15) yields : 

Li.— (nLi,nL2)= 5^hk.iJF[n,m] 
(k,I)eS 

-Li ^ khk,iLi.— (;iLi./iL2) 

(k.l)6S 

-L2 lhk,lLl.T-(«Ll-'»L2) (16) 

(k,l)ES ^ 

Identifying the members of the relation (16) then yields three constraints for h : 

5 Shu=0 
(ka)6S 

Zkhtl=-1 
(k,l)eS 

(k,l)eS 

The chosen filters, which satisfy these conditions, are the following : 

(a) x-dimension gradient filter : 
10 -1/8 0 1/8 

-1/4 0 1/4 
-1/8 0 1/8 

(b) y-dimension gradient filter : 
-1/8 -1/4 -1/8 

15 0 0 0 

1/8 1/4 1/8 
which yields a good approximation of the discrete gradient. 

Different improvements can be proposed for the implementation of the 
algorithm described above, especially in terms of accuracy of motion vector and speed. The 

20 speed of the algorithm can be increased by making it more rapidly convergent. ITie 

initialization step is thus crucial for the convergence speed. A good initialization has to take 
into account as much as possible the previously computed motion vectors. However, the 
assumption about the smoothness of the distribution of the motion vectors over the image 
plane can be untrue (for certain types of motions, for instance when objects in the plane are 

25 moving in different directions) at tiie boundaries of the objects. The algorithm has to be able 
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to detect this event, and to correct the initialization by breaking the smoothness of the motion 
vector field. It is possible to introduce a break test at the end of the initialization step (and 
even after each update of the motion vector). One has to compute two test values: 
Eo(m) = |Bj^(m).Aj/fm)l 

5 Ej,s(m) = |Bj^(m) - Aj,s(fm - fdi) |, 

where Eo is the error without motion (dj = 0), and Ej^ is the error when the computed motion 
vector is taken into account The principle is that if Ej> is greater than Eo, the motion vector 
has to be reinitialized to zero. However, too frequent reinitializations will prevent the 
algorithm from converging. When the prediction error is very small, this case could appear 

10 quite often. That is why a tolerance in this test is introduced, which is defined through 
inequality : 

Ei^>Eo + THR (17) 
where THR is a threshold, to be determined according to the value of the subband 
coefficients. 

15 Another improvement relates to the scanning order. Hie image being a table, it 

is scanned from the first item (i.e. m = (0,0)) to the last one (m=xmax, ymax), as shown in 
Fig. 17. This initialization ovdsr, combined with the initialization neighbourhood, yields a bad 
motion estimation in the lower left comer of the image (since, for the first column of the 
image, the initialization neighbourhood is only reduced to the pixel above). The method is 

20 then not robust and leads to an error drift. Hie scanning order presented in Fig.18 yields 

better results. The image is scanned, the even lines from the left to the right, and the odd lines 
from the right to the left. This aHows an improved causality in the initialization. Indeed, 
every pixel depends on all the previously scanned pixels. The error drift observed in the 
lower left comer of the image does not exist with this method. A better initialization makes 

25 possible for the algorithm to converge faster. The error drift was due to the fact tiiat the 

number of iterations is limited : if the initialization is not good, the algorithm has not enough 
iterations to converge to the right motion vector. Given this new scanning order, the 
initialization neighbourhood has to be changed. For the first line, the neighbourhood is 
reduced to the previous pixel : 

30 *joni* 

For the even lines, the neighbourhood remains unchanged: 

* jl j2 j3 * 

* jo m * * 

Ha ^ % « « 
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A third improvement relates to the approximation subband SB Ajnmx,o of the 
image, which has, by definition, very few details. Thus a pel-recursive motion estimation is 
sometimes not accurate enough. Small errors in the compensated approximation yield bad 
results for the image reconstructed from the compensated subbands. Better results are 
5 obtained if a fiill search is performed for each pixel of Bjinax.o in a window of Ajniax.o, but this 
method is not as fast as the pel-recursive algorithm. The size of the search window is : 

(2.(2^"")+l) by (2.(2'"") + 1) 
as shown in Fig.l9. SB(3,0) being well estimated, it is possible to use its motion vector to 
initialize the other subband SB. If one calls dappioximatitm the motion vector in SB(3,0) at pixel 
10 m(m,n), and do the initialization vector taking into account only spatial neighbours, a new 
initialization vector can be computed through : 

Do = a . dapproximation + (1 - a) . do, (18) 

where a is a constant that allows to control the influence of dappiaximatioii in the initialization. If 
a - 0, dapproximatioo has no influence at ail. Hie same principle can be used for the other 

IS subbands SB. It is then possible to use the previously computed motion vector, at a higher 
level of decomposition j, in order to initialize the algorithm. If one calls dj4.i tiie motion 
vector estimated at level j+1, flie new initialization vector is : D© = a . dj+i + (1 - a) . do. 
The generic algorithm described above can be applied between two subbands. However, if it 
is used, the correlation that exists between the subbands of a same level of decomposition is 

20 not taken into account : die computed motion vector should be the same for each pixel in the 
three subbands (or four when LLrsubband is taken into account). It is then possible to decide 
to estimate, for a given level of decomposition j € {1,2,... jmax}, for each pixel m = (m,n), 
the motion vector d|j (m) : 

Bj^(m)=Ajs (m - dj) for s e {0,... 4} (19) 

25 where : 

. Ao,o represents the first image, 
. Bo.o represents the second image, 

. Aj,s represents the subband number s at the j-th decomposition level of the frame A : 
the subband 0 represents the approximation subband (SB), 
30 the subband 1 represents tiie horizontal detail SB, 
the subband 2 represents tiie vertical detail SB, 
the subband 3 represents tiie diagonal detail SB. 

The assumption of d being smoothly distributed over the subband planes has 
to be made just as previously. 
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As previously, the iterative procedure, carried out for each level of 
decomposition j e {1,2,..- jmax}, comprises four steps including the initialization, the 
computation of an update vector, the update of the motion vector and a break test, but with 
some slight modifications : 
5 The iterative procedure is initialized by the mean value of the motion vector in a causal 
neighbourhood S, 

with for example a typical neighbouriiood of 4 pixels (the motion vectors are supposed to be 
zero out of the image boundaries): 
10 * ki k2 k3 ♦ 

* ko m * * 

:¥ ^ ^ * 4c 

2. Then an update vector uj^ is determined : 

= ( i(VAj,sVA]:3) + ^r^ (- iEj,3(m)VAj,,) (21) 
s=0,l s=0.1 



15 where : 



dm 
dn 



(22) 



is the gradient of the image at point m - d^^ and : 

20 Ej^ (m) = Bj^ (m) - Aj^ (m - dj^) (23) 

s varying from 1 to 3 or from 0 to 3, depending on the number of subbands that have to be 
taken into account (4 for the deeper decomposition level, 3 for the others). 

3. Then the motion vector d[j, i+i is updated : 

25 di.i+, = dj.i + Uj.i (24) 

4. The break test ends die procedure if dy is close enough to dj, this test being for example 
the estimation of the ratio : 

Il2 



Rj.i = 



Kt 



(25) 
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which has to be superior than e (e to be determined). An additional condition is put on the 
maximum number of iterations. 

For the update vector computation, at the first order, the relation (1) becomes, 

for every s : 

Bj^(m) = Aj^(m-dj^) - « JVAj^^ (26) 
where VA j g is the gradient from Eq. (4) and &ij= dj - djj. 
As it is wanted to minimize the square error on all the subbands at the same time: 



;S(Bj.s(m)-Aj^3(m-dj,i) + ^7vAj,,)2 + A||&ijf 
s=0,l 



Ji 



(27) 



the minimum is obtained when 



i.e. : 



E((Bj.s(ni)-Aj,3(m.dj,i)+c5dJ'VAj,3)VAj^3 +Aadj) = 0 (28) 
«=0,1 



15 The relation (28) then becomes : 



( 2(VAj,sVAj3)+^.aij =-Ej,3(m).VAj. s 
y=0,l 



(29) 



Even without regularization, the matrix Xs=:0 i^^h^^^Ts '^^^ ^ 

invertible. It is a 2 by 2 matrix, and the inverted matrix can be computed explicitely. In this 
20 case, the regularization teim >1 1 is required only for the control of the motion vector field 
coherency. 

In case this pel-recursive motion estimation algorithm should be applied to 
the redundant subband, the previous algorithm must be adapted Let f be the scale factor 
caracterizing the ratio between the size of the redundant subband and the size of the 
25 subsampled subband : 

f=2^ 
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where j is the decomposition level. 

It is known that a redundant subband presents the lack translation invaxiance. 
As a result, the pel-recursive motion estimation can be performed between a redundant 
subband (reference frame) and a subsampled subband (current frame). The motion vectors 
5 are then (assuming that even samples have been retained for the subsampled decomposition 
of Bj^) defined through : 

Bj^(m) = Aj^(fm-fdj) (30) 
The iterative procedure is then slightly different : 
1. do(m) is initialized through the relation (20). 
10 2. The update vector Ui remains unchanged, with the new error : 

Ej^ (m) = Bj.s (m) - Aj^ (fm - f di) (31) 
and the new gradient : 



15 



9m 
dn 



(32) 



3. The update equation is unchanged. 

4. The break test remains unchanged. 
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1 . A video coding method based on a subband coding scheme applied to a 
sequence of two-dimensional frames, said method comprising a subband decomposition step 
of a current frame (F2), a motion prediction step, carried out with respect to a previous or 
reference frame (Fl), and a coding step, said method being moreover characterized in that 
said motion prediction step is based on a redundant decomposition of the reference frame, 
according to the following procedure : 

(1) in the approximation subband of said decomposition, the motion is 
estimated by means of a full search operation involving said reference frame and applied to 
each picture element (pixel) of the current frame ; 

(2) for the other subbands, the motion is estimated by means of a pixel (or pel) 
recursive motion estimation algorithm using for the prediction the reference redundant 
subband reconstmcted from the subsampled subbands yielded by said subband 
decomposition. 

2. A coding method according to claim 1, in which said pel-recursive motion 
estimation algorithm, intended to compute the optic flow between two matrices A and B 
representing two successive subbands i.e. to estimate for each pixel m its motion vector d(m) 
in such a manner that B(m) = A(m-d), itself comprises the following sub-steps : 

(a) an initialization sub-step, the iterative algorithm being initialized with the 
mean value do (m) of the motion vector in a causal neighbourhood S : 

the following notations being used : 

m = pixel for which the motion has to be estimated ; 
d(m) = its motion vector ; 

(b) an updating sub-step, for updating di according to the 

relation : 

dj+i = di + Ui, where Ui, detennined at each iteration i, is an update vector given by : 
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where V A is the gradient of the image at the point m-di, E(m) = B(m) - A(m-di) is the 
ptediction error at the current iteration, and /L is a regularization parameter ; 

(c) a decision sub-step, for carrying a break test and thus ending the procedure 
5 if di is close enough to d. 

3. A coding method according to claim 2, in which a typical neighbourhood of 

four pixels is used. 

10 4. . A coding method according to anyone of claims 2 and 3, in which said break 

test is based on the estimation of the following i^o : 

said ratio having to be greater than a given threshold e , 

15 5. A coding method according to claim 4, in which said threshold £ is turned to 

balance the vector detennination precision and the computational load. 

6. A coding method according to anyone of claims 4 and 5, in which said break 
test includes an additional condition put on a maximum number of iterations. 

20 

7. A coding method according to anyone of claims 4 lo 6, in which 

the determination of said update vector Ui is based on a computation step including the 
following minimizing operation : knowing that B(m) = A(m-di) - Si .VA where Si -VA is 
the inner product of VA and vector 5i = d - di, to minimize the square error J : 

25 J = ((B(m) - A(m-di) + Sd^^7Af + ^af 

with respect to <5a , the term >l|<5af being a regularization term provided in view of a trade- 
off between the smoothness of the resulting motion vector field, for large values of A , and 
the accuracy of the motion vectors, for small values of A , the minimum being obtained for 
dl/d3d = 0 and leading to the update vector : 
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8d = Ui=-J50-.VA 



|VA|^+X 



8. A coding method according to claim 7, in which the scanning order is carried 
out in the following way : even lines from the left to the right, odd lines from the right to the 

5 left, the initialization neighbourhood being then also modified for the first line, where it is 
reduced to the previous pixel, and for the odd lines, whete it is inverted to preserve causality 

9. A coding method according to claim 7, in which said motion estimation 
algorithm is re-initialized by testing a break condition with some tolerance, the prediction 

10 error when using the estimated motion vector being, for said test, compaied to the prediction 
error with no motion estunation, plus a threshold. 

10. A coding method according to claim 2, in which the initialization of the 
motion vector is done with a linear combination of the average over the neighbourhood S and 

15 the motion vector obtained from the subband with the same orientation, at the previous 
decomposition level. 

11- A coding method according to claim 1, in which said pel-recursive motion 

estimation algorithm, intended to xise the correlation that exists between the subbands of a 

20 same level of decomposition i.e. to estimate for a given level of decomposition j g { 1,2, , 

jmax} and for each pixel m its motion vector dj(m) in such a manner that Bj;j(m) = Aj^(m-dj) 
for s = {0,..., 4}, the two matrices A and B representing two successive subbands, itself 
comprises the following sub-steps : 

(a) an initialization sub-step, the iterative algorithm being initialized with the 

25 mean value dj (m) of the motion vector in a causal neighbouriiood S : 



Card(S)^S 

the following notations being used : 
k € s(l to 4); 

m = pixel for which the motion has to be estimated ; 
30 dj.o(m) = its motion vector ; 

(b) an updating sub-step, for updating dj, i+i according to the 

relation : 



10 
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M = + Uj,i, where Uj4 detennined at each iteration i is an update vector given by : 

=( i(VAj,,VAj;s) + ^)-^ (- iEj.3(m)VAj.s) 
s=04 s=0,l 



where ; 



dm 
dn 



is the gradient of the image at point m - dj.i and : 
Ei.s (m) = Bj,s (m) ^ Aj^ (m - djj) 

(c) a decision sub-step, for carrying a break test and thus ending the procedure 
if djj is close enough to dj. 

12. A coding method according to claim 1 1, in which a typical neighbourhood of 

four pixels is used. 



13. A coding method according to anyone of claims 11 and 12, in which said 

15 break test is based on the estimation of the following ratio : 

\\2 



J>Ml 

said ratio having to be greater than a given threshold e . 

14. A coding method according to claim 13, in which said break test includes an 
20 additional condition put on a maximum number of iterations. 

15. A coding method according to anyone of claims 13 and 14, in which 
the determination of said update vector Ui is based on a computation step including the 

following minimizing operation : knowing that Bj.s = Aj,s(m-dj,i) - ^SlJ^VAj^g where 

25 Sd^ .VA is the inner product of VA j § and the vector 
&i j = dj - djj, to minimize the square error : 
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SCBj.s(m)-Aj.s(m-dj,i) + (yJVAj.s)2+JAlj 
s=0.1 



Ji 



(27) 



with respect to <Sij , the term j| being a regularization term provided in view of a trade- 
off between the smoothness of the resulting motion vector field, for large values of ^1 , and 
5 the accuracy of the motion vectors, for small values of /I , and the minimum being obtained 
for ajj/a&ij=0. 



wo 03/001814 



PCT/1B02/02362 




wo 03/(Mn814 



PCT/IB«2/02362 



2/10 



M3 


LH3 


LH2 


LHl 


HL3 


HH3 


HL2 


HH2 


HLl 


KH1 



FIG.2 




22 32 

Mtttl .fen >uL2ln] 



iM HH(z)K i2>H7tt11 
24'' 34' 25' ^35 '-26 ^36 




FIG.3 



wo 03/001814 



PCT/IB02/02362 



3/10 



ORS 



HFC-^ 




F\GA 




FIG. 5 



4./10 




FIG.6 



22> 32-. 23\ 23s 





RSB2^ ^RSB3 
RG.7 



wo 03/001814 



PCT/IB02/02362 



6/10 



1 LEVELI /[n] 



Hl[n] 




LH(z) 



FIG.10 



2LEVELSI /Cn] 



Hg*S-*(i^H2[n] 



nan 



slevelsi /M 



FIG.12 



WO03/«(H814 PCT/IB02/02362 



7/10 




FIG.13 




wo 03/001814 



PCT/IB02/02362 



8/10 




O O CJ u 


o 


O CJ o u> 


o o o o 




o o o «-> 






c 

I 

•♦-» 





O-a Ot3 








co-o 






<->-o o-o ono 




CO-O 1X3-^ <0*Q 




0"0 O-O OTD 




ro-o cu-o <D-o 



<T3 fo CO ro 
*o *n IP ^ 

CD fti na 

CO CU CO CP 



O 
CT 



^ I 



C $ o 



wo 03/<)()1814 



PCT/1B02/02362 




FIG.16 



vR)Sition(f(n+V(m-.l) 



a 



2,(2jmaX)^l 



Position (/.n;/.m) 



2.(2J"^^)+1 



FIG.19 



wo 03/001814 



PCT/1B02/02362 



10/10 



FIG.17 



































H 




























— 





FIG.18 



INTERNATIONAL SEARCH REPORT 



In Uonal Application No 

PCT/IB 02/02362 



A. CLASSIFICATION OF SUBJECT MATTER . ^ , _ 

IPC 7 H04N7/26 H04N7/50 H04N7/36 



According to IntenuJIonal Patent gasslHcalion (IPC) or to both national clasaBlcallon and IPC 



B. FIELDS SEARCHED 



Minimum documentation searched (ciaseiflcation system followed by clasBificatlon symbols) 

IPC 7 H04N 



Documentation searched other than minimum documentation to the extent that such documents are hcluded in the fields searched 



Electronic data base oonsutled during the Intemalional search (name of data base and. whore praoUcal. search terms used) 

EPO-Internal 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Galegofy ' Citation of documem. wth htficatlon. wheie appropriate, of the relevant passages 



Relevant to claim No. 



XUGUANG YANG ET AL: "Hierarchical 
backward motion compensation for wavelet 
video coding using optimized Interpolation 
filters" 

IMAGE PROCESSING, 1997. PROCEEDINGS., 
INTERNATIONAL CONFERENCE ON SANTA BARBARA, 
CA, USA 26-29 OCT. 1997, LOS ALAMITOS, CA, 
USA, IEEE COHPUT. SOC, US, 
26 October 1997 (1997-10-26), pages 
85-88, XP010254114 
ISBN: 0-8186-8183-7 
page 87, paragraph 3 



1-15 



m 



Further documents are listed in the continuation of box C. 



ID 



Patent family members are listed in annex 



* Special categories of dted documents : 

'A' document defining the general state of the art which is not 

considered to be of particular rslevance 
*E* earlier document but published on or efter the international 

tSlng dale 

*L* document which may thmw doubts on prtorfty claim(8)or 
whfch is c&ed to estabflsh the pubOcatbn date of another 
citation or other special reason (as spectffsd) 

'O* document referrtig to an oral disclosure, use, exhlbifion or 
other means 

*P document published prior to the international filing date but 
later than the priortiy date claimed 



T bter document published after the international filing date 
or priority date and not In conflict with the application but 
died to understand the principle or theory undert/lng the 

invention 

*X* document of particular relevance; the claimed Invention 
canrK>t be considered novel or cannot be considered to 
Involve an inventive step when the document Is taken alone 

*Y* document of parflcuiar relevance: the dahied invention 
cannot be oonaideTed to Involve an Inventive step when the 
document Is combined with nne or more other such docu- 
ments, such combination being obvious to a person sldiled 
In the art. 

document member of the same patent family 



Date of the actual connpl^n of the international search 



19 September 2002 



Date ot mailing of the International search report 



26/09/2002 



Name and maiflng address of the ISA 

European Patent Office, P.B. 5816 Patenllaan 2 
NL-22B0HVRI|SWi|k 
TeL (431-70) 340-2040. Tx. 31 651 epo nt, 
Fax: (4^1-70) 340-3016 



Authorized officer 



Luckett, P 



Foim PCT/ISA/210 (saoond shesl) (JuV 1992} 



INTERNATIONAL SEARCH REPORT 



In tlonal Appitcation No 

PCT/IB 02/02362 



C.(Continuatlan) DOCUMENTS CONStOERED TO BE RELEVANT 



Category • Cflatlon of document, with Indication .wtiere appropriate, of the relevant passages 



netsvarH to claim No. 



BOEROCZKY L ET AL: "MOTIONS ESTIMATION 
FOR WAVELET TRANSFORM CODING" 
JOURNAL ON COMMUNICATIONS, PROFINFO, 
BUDAPEST, HU, 

vol. 45, May 1994 (1994-05), pages 25-27, 

XP001028196 

ISSN: 0866-5583 

page 26, paragraph 3.1 

US 5 253 058 A (GHARAVI HANID) 
12 October 1993 (1993-10-12) 
column 8, line 35 -column 8, line 66; 
figure 4 

US 4 969 040 A (GHARAVI HAMID) 
6 November 1990 (1990-11-06) 
column 7, line 66 -column 8, line 14 

WO 00 13420 A (PIRIM PATRICK ;H0LDING B E 
V SA (LU)) 9 March 2000 (2000-03-09) 
claim 44 



1-15 



1-15 



1-15 



1-15 



Fom PCT/ISMI10 (canllnaallon ol annd ahaal) (Jiiy isge) 



INTERNATIONAL SEARCH REPORT 
„ Jormailon on patent family members 



Inr lonal Application No 

PCT/IB 02/02362 



Patent document 


PublicaUon 


Patent family 


Pubfication 


cited In search report 


date 


member(s) 


date 



us 52530B8 



12-10-1993 WO 



9320653 Al 



14-10-1993 



US 4969040 A 06-11-1990 NONE 



UO 0013420 


A 


09-03-2000 FR 


2782876 Al 


03-03-2000 




FR 


2782878 Al 


03-03-2000 






AU 


5741799 A 


21-03-2000 






UO 


0013420 A2 


09-03-2000 






EP 


1110408 A2 


27-06-2001 



Fonn PCT/ISA/210 (patent lamify aimax) (July 1962} 



